1. Field of the Invention
The present invention relates to an information processing apparatus and a control method thereof, and more particularly to measuring the position and orientation of an image capturing apparatus and an object using images.
2. Description of the Related Art
In recent years, much research has been carried out into augmented reality (hereinafter, AR) technology for displaying information in virtual space superimposed on real space. A video see-through head mounted display (hereinafter, HMD) is a typical example of an information presentation apparatuses for use with AR technology. A camera for capturing real space is built into the video see-through HMD, and the HMD draws a virtual object using computer graphics, according to the position and orientation of the camera in real space. A composite image obtained by further superimposing the drawn virtual object on the image in real space is displayed on an HMD display device such as a liquid crystal panel. This information presentation apparatus allows the user to experience a virtual object as though it exists in real space.
“Alignment” is one of the major problems that must be solved in realizing AR technology. Alignment in AR involves achieving geometric consistency between a virtual object and real space. In order for the user to experience a virtual object as though it exists in real space, geometric alignment needs to be performed correctly. In other words, the virtual object must be presented to the user as always existing where it would be in real space. In the case of using a video see-through HMD, this alignment involves measuring the position and orientation of the camera built into the HMD in a coordinate system set in the real space. The position and orientation of the camera can be measured using a physical sensor for measuring camera position and orientation with six degrees of freedom such as a magnetic sensor or an ultrasonic sensor, for example.
Incidentally, if a video see-through HMD is used, image information from the camera built into the HMD can be used to measure camera position and orientation. Such a measurement method using image information is widely used because of the simplicity and low cost with which measurement can be performed compared with a method using physical sensors.
A Non-Patent Document 1 discloses a method of measuring camera position and orientation using a three-dimensional geometric model describing real space and physical objects by line segments, and edge information on images captured by the camera. Here, an “edge” denotes a point on an image where the density rapidly changes. With this method, camera position and orientation are calculated by positioning the numerous edges detected in an image so as to correspond to the line segments of a three-dimensional geometric model projected onto the image based on the position and orientation of the camera. More specifically, the line segments of the three-dimensional geometric model are virtually projected onto the image based on predicted values of the camera position and orientation input in advance, and edge detection is performed in proximity to the projected line segments. Camera position and orientation are then calculated by repeatedly correcting the predicted values of camera position and orientation, so that the detected edges lie on the projected line segments.
With the aforementioned method of measuring camera position and orientation using edges, the position and orientation of the camera in the previous frame are primarily used as the predicted values of camera position and orientation. Thus, if measurement of position and orientation in a given frame fails, measurement of position and orientation in the following frame cannot be performed. Measurement failure occurs if the camera is moved quickly, for example. In such a case, measurement of position and orientation tends to fail because of edges not being detected due to motion blur occurring in the image, or edges no longer being in proximity to the line segments projected based on the predicted values of camera position and orientation. Also, with the initial frame, the camera position and orientation in a previous frame cannot be obtained.
Thus, in order to actually use the method of measuring camera position and orientation using edges, an initialization process for measuring camera position and orientation without using predicted values of camera position and orientation is required for the initial frame and for the frame immediately following measurement failure.
One such measure involves firstly setting a predetermined default position and default orientation, and moving the camera so as to be in the default position and default orientation, and then performing initialization of camera position and orientation using edges, with the default position and default orientation as predicted values.
Another measure involves measuring the camera position and orientation from features detected in an image, without using predicted values of the camera position and orientation. For example, a Non-Patent Document 2 discloses a method of measuring camera position and orientation based on correspondences between straight lines detected in an image and straight lines corresponding thereto in three-dimensional space. With this method, camera position and orientation are calculated by solving linear equations based on eight or more line correspondences. With the method of Non-Patent Document 2, the method of matching straight lines detected in an image and straight lines corresponding thereto in three-dimensional space is not indicated. Thus, in order to measure camera position and orientation using the method disclosed in Non-Patent Document 2 from a state of not knowing the method of matching straight lines in an image and straight lines in three-dimensional space, it is necessary to derive correspondences between the straight lines detected in the image and the straight lines in three-dimensional space using some sort of measure. Generally, camera position and orientation are calculated by randomly matching straight lines in an image and straight lines in three-dimensional space, and the result calculated when using correspondences having the highest consistency is output as the final camera position and orientation.
Also, with a Patent Document 1, stable position and orientation measurement is performed, by mounting an inclination sensor to the camera, and measuring camera position and orientation using a combination of values measured by the inclination sensor and image information (point information). Also, with a Patent Document 2, the azimuth angle of the camera is calculated, by mounting an inclination sensor to the camera, and using values measured by the inclination sensor and information on straight lines in the image.
Since the aforementioned initialization process generally requires time, the camera position and orientation in an image used in the initialization process may, in the case where the camera is constantly moving, for example, differ greatly from the actual position and orientation of the camera at the point in time at which the initialization process ends. In such a case, position and orientation measurement will likely fail, given that edges no longer exist in proximity to the line segments projected based on the values of the camera position and orientation after the initialization process has ended. In a Patent Document 3, tracking of position and orientation information on the camera or an object is enabled by holding the image data of frames captured by the camera, and sequentially performing processing, from the image of the next frame after the image used in the initialization process until the current image.